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[Document Name] Specification 

[Title of the Invention] Image Input Camera 

[Scope of Claims for Patent] 

[Claim l] An image input camera, comprising: 

input means for inputting image data of an object; 

calculating means for calculating, from a plurality of pieces of the 
image data input by the input means from multiple viewpoints, a 
coordinate conversion parameter for patching up the plurality of pieces of 
the image data; and 

patch up means for patching up the plurality of pieces of the image 
data by the parameter obtained by the calculating means. 

[Claim 2] The image input camera according to claim 1, wherein 
the calculating means obtains, as the coordinate conversion parameters for 
patching up the plurality of pieces of the image data, a rotation angle of the 
object relative to the input means and a position of a rotation axis of the 
input means, based on the plurality of pieces of the image data input. 

[Claim 3] The image input camera according to claim 1, wherein 
the calculating means obtains, as the coordinate conversion parameters for 
patching up the plurality of pieces of the image data, a rotation angle of the 
object relative to the input means and a position of a rotation axis of the 
object, based on the plurality of pieces of the image data input. 

[Claim 4] The image input camera according to any of claims 1-3, 
wherein the patch up means performs coordinate conversion of the 
plurality of pieces of the image data to one coordinate system, based on the 
coordinate conversion parameter obtained by the calculating means. 
[Detailed Description of the Invention] 

[Technical Field to Which the Invention Belongs] 

The present invention relates to an image input camera that 
generates image data of an object having a relatively complex shape, and 
performs image patch up using the image data. 

[Prior Art] 

Conventionally, among means for recognizing a three-dimensional 
shape of an object, for example, a light-section method has often been used 
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as most practical means. With the light- section method, as shown in Fig. 
1, an object is irradiated with a slit shaped laser light S, and a slit image of 
the object 1 corresponding to the slit shaped light S is obtained on the 
imaging plane of the camera. A spatial coordinate of a point p on the 
object corresponding to a point p' on the slit is calculated as the coordinate 
of a crossing point of a plane S formed by the slit shaped light and a 
straight line L connecting the point p* and the center O of the lens of the 
imaging device. As such, spatial coordinates of points on the object surface 
corresponding to the respective points on the slit image are obtained from 
an image, and three-dimensional information of the entire object is 
obtained by repeating image input while moving the slit shaped light in a 
horizontal direction. 

A plurality of pieces of image data obtained by the above-described 
method have conventionally been patched up in accordance with conversion 
parameters calculated based on the position of the camera or the position of 
the rotary stage detected with mechanically high precision. 

[Problems to be Solved by the Invention] 

With the conventional patch up method of a plurality of pieces of 
image data, it is necessary to detect the position of the camera or the rotary 
stage with mechanically high precision. To achieve such mechanically 
high precision, however, the apparatus would be extremely expensive. 
Further, it would be difficult to use the same in such a rough input 
environment that an input is made with a hand-held camera or the like. 

The present invention has been made to eliminate the deficiencies of 
the conventional art, and its object is to perform image patch up at a 
relatively low cost, by inputting a plurality of pieces of image data via 
multiple viewpoints* calculating relative positions of the input images, and 
obtaining a conversion parameter of the image data to perform the patch up 
thereof. 

[Means for Solving the Problems] 

To achieve the above-described object, an image input camera of the 
present invention includes input means for inputting image data of an 
object, means for calculating, from a plurality of pieces of the image data 
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input from multiple viewpoints, a coordinate conversion parameter for 
patching up the plurality of pieces of the image data, and means for 
patching up the plurality of pieces of the image data. 
[Function] 

According to the configuration of the present invention, relative 
positions of the images are calculated for the plurality of pieces of image 
data having been input, a coordinate conversion parameter is obtained 
therefrom, and coordinate conversion is carried out to perform the patch up 
processing. 

[Embodiments] 

Hereinafter, embodiments of the present invention will be described 
with reference to the drawings. 

Firstly, Fig. 2 is a schematic block diagram of the entire apparatus in 
accordance with the present invention. Briefly stated, the apparatus of 
the present invention includes a light projecting optical system 2 for 
irradiating an object 1 with laser beam, which is output from a 
semiconductor laser 5 and turned into a slit shaped light, and a light 
receiving optical system 3 for guiding the projected laser beam to imaging 
sensors 24 and 12. These optical systems are arranged on a same rotary 
frame 4. In addition to the optical systems, the apparatus includes a 
signal processing system for processing a signal output from a sensor for 
generating pitch-shifted images and color images, and a recording device 
for recording the generated images. In Fig. 2, solid arrows denote flow of 
electric signals such as image signals, control signals and so on, while 
dotted arrows denote the flow of projected light. Details of these optical 
systems will be given later. 

An outline of the signal processing system will be described. With 
respect to an image obtained by a distance image sensor 12, subtraction 
between an image 18a when slit shaped light is projected and an image 18b 
when slit shaped light is not projected is performed, and calculation of the 
position of centroid of the incident light 19, calculation of pitch-shift 
information 20 and pitch-shift image generating process 21 are performed 
on the image. The obtained pitch-shifted image is utilized as an output to 
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an output terminal 50 after NTSC conversion 27, or as digital information 
to be transferred to an SCSI terminal 49 or an internal recording device 22. 
The image obtained by a color image sensor 24 is subjected to analog 
processing 25 and then to color image generating process 26. The 
resulting color image is utilized as an output to an output terminal 51 after 
NTSC conversion 28, or as digital information to be transferred to SCSI 
terminal 49 or recording device 22. 

Fig. 3 is a perspective view showing the schematic structure of the 
entire apparatus. 

In this embodiment, a generating system for distance image having 
256 points of distance information in the lengthwise direction of the slit 
shaped light and 324 points of distance information in the scanning 
direction of the slit will be described as an example. An LCD monitor 41 
provides display of a color image formed by color image sensor 24, three- 
dimensional data stored in an internal or external recording device, various 
other information, menu for selection, and so on. A cursor key 42, a select 
key 43 and a cancel key 44 are operating members for setting, for example, 
various modes from the menu, or for selecting images. A zoom button 45 is 
provided for changing focal length of the light projecting/light receiving 
optical systems. An MF button 46 is for manual focusing. A shutter 
button 47 is for taking a distance image when turned ON in a shutter mode, 
which will be described later. A drive 48 such as an internal magnet" optic 
disc (hereinafter referred to as MO), a mini disc (hereinafter referred to as 
MD) is provided as the storage apparatus for the picked up image. A 
terminal 49 is, for example, an SCSI terminal for digital input/output of 
signals of images and the like. A pitched-shifted image output terminal 50 
and a color image output terminal 51 are provided for outputting images in 
the form of analog signals, and the images are provided as NTSC video 
signals, for example. 

The light projecting optical system scans the object by moving a 
horizontally elongate slit shaped light in upward and downward directions, 
and the light beam from semiconductor laser 5 is directed to the object 
through a rotating polygon mirror 7, a condenser lens 10, a light directing 
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zoom lens 11 and so on. The light receiving optical system picks up an 
image by means of a light receiving zoom lens 14, a beam splitter 15 and so 
on, and further by distance image sensor 12 and color image sensor 24 
arranged on a light receiving image pickup plane. Details of the optical 
systems and the imaging system will be given later. 

The slit shaped light from the light projecting system is moved 
downward one pixel pitch by one pixel pitch of the distance image sensor 12, 
by means of constantly rotating polygon mirror 7, while distance image 
sensor 12 accumulates one image. The distance image sensor scans the 
accumulated image information, provides an output, and performs 
accumulation of the next image. From the image provided at one input, 
distance information of 256 points in the lengthwise direction of the slit 
shaped light can be calculated. Further, by repeating the mirror scanning 
and taking of images 324 times, a distance image consisting of 256 x 324 
points is generated. 

As for the distance range to the object measured by one slit shaped 
light, the minimum and maximum measurement distances are limited, and 
therefore the range of incident light which is the slit shaped light reflected 
by the object and entering the image pickup device is limited within a 
certain range. This is because the light projecting system and the light, 
receiving system are arranged apart from each other by a base length 
(length: l). This is illustrated in Fig. 4 in which the Z axis represents a 
direction perpendicular to the image pickup plane for the distance image. 
The position of the dotted line d is a reference plane for measurement, and. 
the distance from the plane of the device corresponds to d. 

Df represents maximum distance for measurement and Dri 
represents minimum distance for measurement. Now, if the plane cut by 
the slit shaped light projected from the light projecting system is slit A (SA), 
the scope on the plane of the image pickup device receiving the slit shaped 
light reflected by the surface of the object is limited to a closed area Ar, in 
which a position of projection on the image pickup device of the three- 
dimensional position of an intersection PAn between the minimum distance 
Dn for measurement and the slit A is the lowermost point in the figure, and 
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the projected point on the image pickup device of the three-dimensional 
position of the intersection Paf between the maximum distance Df for 
measurement and slit A, projected on the image pickup device with the 
position of the main point of the image pickup system being the center, is 
the uppermost point in the figure. Assuming that the light projecting 
system and the light receiving system have the same positional relation, in 
the case of slit B (SB), the scope on the plane of the image pickup device is 
limited to a closed area Br on the image pickup device, in which the point of 
projection of the intersection PBn of the minimum distance for 
measurement Dn and slit B is the lowermost position in the figure, and the 
point of projection of intersection Pbf of the maximum distance for 
measurement Df and the slit B is the uppermost point in the figure. 

Therefore, in the measuring apparatus, the position of the centroid of 
the laser beam received at 256 lines is calculated based on the input image. 
More specifically, the position of the centroid is calculated as the amount of 
deviation from the reference plane for measurement, that is determined 
based on an object distance output from an auto focus unit and direction of 
the projected slit shaped light, that is, the time from the start of scanning. 
The calculation of the amount of pitch shift will be described with reference 
to Fig. 5. Fig. 5 shows light intensity distribution generated by the slit 
shaped light directed to the object. The sections at the lower portion of the 
figure represent areas monitored by each of the elements of the distance 
image sensor. These sections have numbers 1, 2, 3, 4, ... allotted thereto, 
starting from the front side. A slit shaped light having very narrow slit 
width is moved for scanning only by 1 pitch of the distance image sensor by 
the rotation of polygon mirror 7 while one image is accumulated. 
Therefore, the light intensity distribution when one image is input 
corresponds to a rectangular light intensity distribution of which width 
corresponds to 1 pitch of the distance image sensor. 

In order to calculate distance information in the direction of the Z 
axis for each pixel of the distance image sensor, such a rectangular light 
intensity distribution having the width of 1 pitch is desirable. When the 
width of the light intensity distribution becomes wider than 1 pitch, the 



distance information measured would be calculated as weighted mean of 
the intensity of light received at adjacent areas, and hence correct distance 
information would not be obtained. 

Assume that there is a step-shaped object surface such as 
represented by the dots in Fig. 5, and a slit shaped light is directed from a 
direction vertical to the plane of the object. The thin rectangular 
parallelepiped represents the light intensity distribution of the slit shaped 
light and the hatched area represents the slit- shaped image irradiated by 
the light beam. When we assume a positional relation in which an optical 
axis Oxp of the light receiving optical system is provided inclined to the left 
from an optical axis Oxa of the light projecting system, the light intensity 
distribution of the received slit shaped light at the light receiving plane 
would be as shown in Fig. 6, because of a filter, which will be described 
later. It is desirable to remove fixed light component other than the laser 
beam component so that the fixed light component is not included in the 
received light intensity. For this purpose, an image irradiated with the 
laser beam and an image not irradiated with the laser beam are both input, 
and the difference therebetween is used. The sections at the lower portion 
represent respective element regions of the distance image sensor. 

In front of the distance image sensor, there is provided an anisotropic 
optical filter which does not degrade resolution in the lengthwise direction 
of the received slit shaped light but degrades the resolution in the 
widthwise direction of the slit shaped light, and by means of this filter, the 
light intensity distribution having such a Gaussian distribution as shown 
in Fig. 6 results. With respect to this light intensity distribution, by 
calculating the centroid of the light intensity distribution from respective 
sensors for columns 1, 2, 3, 4, the position at which the light is received 
can be calculated with higher resolution than the pixel pitch. The reason 
why the width of the slit shaped light incident on the sensor is not 
narrowed but selected to have the width of about 5 to 6 pixels by using a 
filter for detecting the position at which the slit shaped light is received is 
that when the width of the incident slit shaped light becomes narrower 
than the width of one pixel, the resolution for detecting the position could 
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be at most the same as the pixel pitch. 

Based on the light intensity distribution Dl received by the first 
column, the position Gl of the centroid of the first column is calculated. In 
the similar manner, the positions G2, G3, G4, ... of centroid of the second, 
third, fourth and the following columns are calculated, and thus the 
centroid of each column is calculated. As shown in the figure, the optical 
axis of the light projecting system is vertical to the plane of the object. 
However, the optical axis of the light receiving system is inclined to the left. 
Therefore, when the object has a step as shown in Fig. 5, the centroid of the 
higher portion (third and fourth columns) is positioned shifted to the right, 
with respect to the centroid of the lower portion (first and second columns). 
Though the distribution Dl of the first column and distribution D4 of the 
fourth column only are shown in Fig. 6, the distribution D2 of the second 
column is the same as the distribution Dl of the first column, and the 
distribution D3 of the third column is the same as the distribution D4 of the 
fourth column. The relation between the light intensity distribution and 
the positions of the centroid is represented two dimensionally in Fig. 7. 
Since the distributions of the first and second columns are the same, the 
calculated centers of gravities Gl and G2 are detected as the same position. 
Similarly, since the distributions of the third and fourth columns are the 
same, the calculated centers of gravities G3 and G4 are detected as the 
same position. 

In this manner, from a slit-shaped image corresponding to one slit, 
positions of the incident light at 256 points are calculated. By performing 
similar calculation for the slits directed to 324 directions, 324 images are 
obtained, and a pitch-shifted image consisting of 256 x 324 points is 
obtained. The obtained pitch-shifted image consists of only the positional 
information of the slit shaped light. Therefore, in order to obtain an 
accurate distance image, calibration (correction) based on a table of detailed 
data such as lens aberration correction is necessary. Therefore, lens 
aberration estimated from the focal length f and in-focus position d of the 
taking lens is calculated, corrected, and distortion in the longitudinal and 
lateral directions with respect to the camera is corrected. Similar 
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operation is performed with respect to the color image. The data necessary 
at that time includes information of various measurement lenses, that is, 
focal length f and in-focus position d. In the system of the present 
embodiment, calibration is performed on a computer system, and 
connection to the measurement apparatus of the present invention (shown 
in Fig. 3) is provided by SCSI terminal, for example. Alternatively, the 
data may be shared by using a recording medium such as MO. 

In this manner, from the body of the measuring apparatus, color 
images and pitch-shifted images are provided as digital signals from a 
terminal such as SCSI terminal, or provided as analog video signals from 
an output terminal such as BTSC terminal. Data necessary for calibration 
are provided to the computer as digital signals from SCSI, for example. 
When a drive 48 such as internal MO or MD is to be used, images and 
various data are recorded on the recording medium. 

The taken pitch-shifted images and color images are transferred to a 
computer connected to the measuring apparatus, together with various 
taking lens information. In the computer, based on the transferred pitch- 
shifted images and the taking lens information, the data are calibrated and 
converted to a distance image having information with respect to the 
distance to the object. As for the pitch-shifted image, after calibration, a 
conversion curve with respect to the stored amount of shifting and 
measured distance is extracted for every XY position, longitudinal and 
lateral positions on the image plane, focal length f and in-focus position d, 
and the pitch-shifted image is converted to a distance image based on the 
relevant conversion curve. 

Conversion to the distance image is well known and the detailed 
explanation is given, for example, in Institute of Electro nics. Information 
and Communication Engineers. Workshop Material PRU 91*113, Onodera 
et al., "Geometrical Correction of Image Without Necessitating Camera 
Positioning", Journal of Institute Electronics, Information and 
Communication Engineers. D-II vol. J74-D-II, No. 0, pp. 1227-1235, '91/9 
Ueshiba, Yoshimi, Oshima et al, "Highly Precise Calibration of a Range 
Finder Based on Three-Dimensional Model of Optical System." 
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Hereinafter, the imaging method and the data processing of the 
present invention will be described in detail. 

Firstly, high-precision input by divisional taking, using a camera 
universal head, in the three-dimensional shape measuring apparatus is 
described. The three-dimensional resolution and precision are determined 
when the distance between the light projection system and the light 
receiving system, i.e., the base length 1, focal length f and a distance d to an 
object to be measured are determined. Thus, the high-precision 
measurement is achieved by elongating the focal length f. In other words, 
the precision in measurement increases in teleside. In such a case, 
however, although a three-dimensional image of high precision in 
measurement may be obtained, the field of view becomes narrower as the 
focal length f become longer. 

Therefore, the focal length f is set to a value corresponding to the 
desired resolution and precision for measurement and the range of the field 
of view is divided into a plurality of regions by operating a rotary frame 4 
such as an electrical universal head. Measurement is performed for every 
divided region, and the resulting images are put together or patched up to 
re -construct one image. By providing such a function, a three-dimensional 
shape measuring apparatus of which resolution can be varied is realized. 
By utilizing this function, environmental measurement becomes possible by 
performing three-dimensional measurement of the entire peripheral space. 
This operation will be described referring to a specific example. The 
example shown in Fig. 8 is a simplified illustration, in which the light 
projecting system 2 and light receiving system 3 are arranged at positions 
having horizontal relation, which is different from the example shown in 
Fig. 3. In this arrangement, the slit shaped light has its length extending 
in the longitudinal direction, and therefore scanning must be carried out in 
left and right directions. 

The manner of operation utilizing the image patch up function is 
shown in Fig. 8. Fig. 9 is a flowchart of the operation utilizing the image 
patch up function. Fig. 10 shows the state of display when this function is 
used, in which there is provided a display portion indicating the precision 
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in measurement below the image display portion. 

Firstly, referring to Fig. 8 (a), the zoom drive system 16 is driven to 
set the range of the field of view to a wide angle state (focal length fO), 
allowing sensing of the object 1 in the range of the field of view, by the 
operation of the user (step #101). The resolution in the Z axis direction 
(see Fig. 4« the direction of the ups and downs of the object) assumed at this 
time is represented by a bar indication below the image, as shown in Fig. 
10 (a). When the base length is fixed as in the present system, briefly, the 
resolution AZ in the direction of the Z axis satisfies the following relation 
between the distance d to the object to be measured and the focal length f at 
the time of measurement* 

AZ = Kxd(d-f)/f ... (1) 

where K is a coefficient for estimating the resolution in the direction of the 
Z axis, which is determined by the sensor pitch and so on. The zooming 
operation described above is performed by transmitting a command from a 
system computer through SCSI terminal. Setting of operations such as 
zooming operation and releasing operation can be made by remote control. 

When the user determines that the above-described setting allows 
measurement with sufficient precision and sufficient resolution (NO in step 
#102), then measurement is started by the releasing operation by the user 
(step #103), and the result is given on the display (step #104). In this 
display, the input pitch -shifted image or color image is displayed, as well as 
the measurement resolution in the direction of the Z axis obtained at that 
time, displayed in the shape of a bar below the image, as shown in Fig. 10 
(a). As a result, if measurement with higher precision is not necessary 
(NO in step #105), measurement is completed, and whether or not the 
obtained result is to be written to a storage medium is determined, and the 
corresponding processing is performed. Thus the operation is completed. 

When the user determines that measurement is not performed with 
sufficient precision (YES in step #102), the user can instruct re- 
measurement with the precision and resolution changed by key operation 
to the desired resolution in the direction of Z axis and desired precision, 
referring to the pitch-shifted image taken by the first releasing operation or 
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referring to the display of the measured resolution in the direction of the Z 
axis (YES in step #105). 

When the key input for setting the precision is entered, the system 
stores the state at that time. More specifically, the system stores the focal 
length fO at which the complete view of the object is obtained, and 
approximate distance d to the object to be measured obtained from the AF 
sensor, and hence stores the scope of the field of view (step #106). Further, 
based on the input desired measurement resolution in the direction of the Z 
axis and the approximate distance d, the system calculates the focal length 
f 1 to be set in accordance with the equation (l) above (step #107). 

When the focal length f 1 is calculated, automatic zooming is 
performed to the focal length fl (step #108); the number of frames to be 
input in divided manner and the angles of panning and tilting are 
calculated based on the stored scope of the field of view to be measured, the 
approximate distance d, and the focal length f 1, and the position of the field 
of view is set by panning and tilting rotary frame (step #109); and 
measurement is performed for every divided input frame (step #110). The 
images input dividedly when the image patch up function is utilized are set 
to include overlapping portions which are used for patching up the divided 
images to re-construct the original one image. 

The obtained pitch-shifted image, color image, information indicative 
of the directions of the field of view in the X and Y directions taken (for 
example, decoded angle values of panning and tilting, order of taking in the 
X and Y directions, and so on), the lens focal length, and information of 
measurement distance are stored in an internal MO storage device (step 
#111). At this time, directory information such as file name, file size and 
so on may not be written to the memory but such directory information may 
be written after confirmation by the user at the last step of operation, so 
that the information is stored temporarily. 

Thereafter, by controlling the field of view to a position of the field of 
view slightly overlapping the position of the field of view of the previous 
operation by panning and tilting in accordance with the calculated angles 
of panning and tilting, the image of the adjacent region is input. By 
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repeating this operation, the images of the entire regions are input (NO in 
step #112, see Fig. 8 (b)). 

Upon completion of the input of the entire region (YES in step #112), 
the initial camera attitude and initial focal length before enhancing the 
precision in measurement are resumed (step #113) and the operation is 
completed. The control waits for the determination of writing by the user. 
When there is a write instruction, directory information is written. If 
there is not a write instruction, the directory information is not written and 
the operation is completed. In that case, the information continuously 
stored in the memory is erased. 

When measurement is performed in advance and thereafter 
measurement is again performed as in the operation of the above example, 
the distance to the object and distribution of the distance in the view angle 
of measurement have been completed by the first measurement. Therefore, 
re-measurement for patching up is not performed for such a divisional 
input frame having large difference from the distance to the object, that is, 
the frame consisting only of the peripheral region (background) different 
from the object, and re-measurement may be performed only for the 
divisional input frames including the object to be measured. In the 
example shown in Fig. 10 (b), the dotted region including the object of 
measurement corresponds to the region for which re-measurement is 
performed. Other regions do not include the object for measurement and 
therefore re-measurement is not performed therefor. 

As described above, high-speed three-dimensional measurement is 
possible, and, by repeating partial inputs and patching up the resulting 
images based on the three-dimensional measurement, three-dimensional 
shape measurement can be performed of which resolution can be set freely. 

In such a patch up measurement, the resolution of the whole image 
frames is uniform. However, there may be an object that requires data of 
high resolution for some portions and low resolution for other portions. 
For example, eyes, mouth and nose of one's face are abound in complex 
shape and color information, while low resolution is sufficient for 
measuring cheeks, forehead and so on. For such an object, patch up of 
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data may be utilized by partial zooming operation, which results in highly 
efficient data input. The partial zooming patch up function is realized by 
the following operations. 

Fig. 11 is a flowchart showing the partial zooming patch up function. 
First, in step #201, setting of the field of view providing the complete view 
of the object is performed, in the similar manner as the uniform resolution 
patch up described above. In step #202, partial zooming input mode is 
selected. When selection is done* presently set values of focal length fO 
and values of decoded angles of panning and tilting are stored (step #203). 
Measurement is started with focal length fO, and image input is provided as 
rough image data (step #204). The pitch-shifted image, color image, 
information indicating the directions of the field of view in the X and Y 
directions at which the image is taken (for example, decoded angle values 
of panning and tilting), the lens focal length, and information of 
measurement distance are stored in an inner storage device (step #205). 
Thereafter, in step #206, zooming is performed to attain the maximum focal 
length fmax, the rough image data mentioned above is analyzed, and 
whether or not re-measurement is to be performed on every divided input 
frames input after zooming is determined. 

When zooming is performed and measurement is done with the 
maximum focal length fmax, the approximate data is divided to the frame 
size to be input. In step #207, the positions X, Y for panning and tilting 
are set to the start initial positions Xs and Ys. In step #208, panning and 
tilting are controlled to the positions X and Y. Then, in step #209, color 
information, i.e., R, G and B values of the initial input color image of the 
region X ± AX and Y ± AY are subjected to statistical processing, and 
standard deviations aR, aG and aB of respective regions are calculated. In 
step #210, whether all the calculated values of the standard deviations aR, 
aG and aB are within the set previous values are determined. If these are 
within the prescribed values, it is determined that the small area has 
uniform brightness information, and therefore zooming measurement is not 
performed but the flow proceeds to step #211. When any of the standard 
deviations aR, aG and aB exceeds the prescribed value, it is determined 
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that the small region has complicated color information, and therefore 
zooming measurement is performed (step #213). 

In step #211, standard deviation ad is calculated based on the 
information of the initial input distance value d in the region of X ± AX, Y ± 
AY. In step #212, whether the calculated value of the standard deviation 
ad is within a set prescribed value is determined. If it is within the 
prescribed value, it is determined that the small region is a flat region 
having little variation in shape, and therefore zooming measurement is not 
performed but the flow proceeds to step #215. If it exceeds the prescribed 
value, it is determined that the small region has complicated shape 
(distance information), and zooming measurement is performed (step #213). 

After the zooming measurement in step #213, the obtained pitch- 
shifted image, color image, information indicative of the direction of the 
field of view of Z and Y directions at which the image is taken (for example, 
decoded angle values of panning and tilting), lens focal length, information 
of distance for measurement and so on are stored in an internal storage 
device such as MO (step #214). Thereafter, the flow proceeds to step #215. 

In step #215, the panning and tilting position X is changed by 2 X. 
In step #216, whether or not scanning in the X direction is completed is 
determined. If it is not completed, the flow returns to the step #208. If it 
is completed, the panning tilting position Y is changed by 2 Y in step #217. 
In step #218, whether scanning is completed or not is determined, and if it 
is not completed, the flow returns to step #208. If the scanning is 
completed, the flow proceeds to step #219, and this routine terminates. 

In this manner, both the schematic image data and partial detailed 
image information allowing determination of the position can be input. By 
patching up the data of the schematic image and the partial detailed image 
data corresponding to the position, highly efficient three-dimensional input 
corresponding to complexities of the shape and color information can be 
realized. 

In the above-described embodiment, photographing is performed 
with precision corresponding to the shape and color of the object by 
changing the focal length of the lens by zooming. However, it is also 
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possible to use two lenses (lenses of long focal length and short focal length), 
i.e., to use the lens of long focal length for a region where the shape and 
color of the object are determined to be complex and the lens of short focal 
length for the other region. 

The data processing for patch up is now described. 

In this processing, data of a plurality of images input by the above- 
described camera device are coordinate converted into one coordinate 
system to thereby obtain data of one image. 

Firstly, the case where patch up is performed with a camera 
mounted on a universal head is described. In this case, a plurality of 
pictures of a fixed object are taken by panning and tilting the camera 
mounted on the universal head that allows panning and tilting, and the 
data of the plurality of photographed images are converted to one 
coordinate system to obtain a patched up image. 

As described in the "Problems to be Solved by the Invention" above, 
when the camera is to be panned and tilted, highly precise patching up is 
possible without any problem if the angle of rotation can be controlled 
precisely. However, such a highly precise universal head is very expensive, 
and therefore photographing by using a general, not so expensive universal 
head is desired, which universal head may have considerable error in 
controlling the angle of rotation. 

In that case, a camera model such as shown in Fig. 12 is prepared. 
This is a camera which allows panning and tilting, represented in a three- 
dimensional coordinate system. In Fig. 12, the reference character C 
denotes the camera, 0 denotes the axis of rotation of the camera (panning), 
and <D represents the axis of rotation of the camera (tilting). 

Parameters of the model (position and direction of the axis of 
rotation for panning, position and direction of the axis of rotation for 
tilting) are calculated in advance by calibration. Searching of the junction 
point (at which two image data are jointed) carried out subsequently is 
performed by changing parameters 0 (pan angle) and <J> (tilting angle) of 
the model. 

This operation is now described with reference to the flowchart of Fig. 
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13. Firstly, two-dimensional color image, three-dimensional data, focal 
length and the distance to the reference plane are taken from the 
photographed data (stored in the storage device in the camera apparatus, 
as described above) (step #301). Then, the junction point is searched for 
from the two-dimensional color image (step #302: details will be given later). 

However, the points of measurement of the two images photographed 
by panning and tilting the camera do not always coincide with each other 
(even when the images are photographed with the same sensing distance 
and focal length, the images deviate from each other by half pixel, at most). 
Therefore, when the deviation is within 1 pixel, searching of the junction 
point is regarded successful, based on the color image (two-dimensional 
data), and searching hereafter is performed by using the three-dimensional 
data. 

Firstly, based on the junction point of the two-dimensional images, 
the focal length and the distance to the reference plane, the angles of 
camera rotation (pan angle 0 and tilting angle O) are calculated (step #303, 
details will be given later). Thereafter, according to the calculated camera 
rotation angles, coordinate conversion parameters for the three- 
dimensional space are calculated (step #304, details will be given later); 

Then, a square sum of an angle formed by normals of two planes 
passing through the junction portion is regarded as an evaluation value, 
and search for minimizing the evaluation value is performed (step #305). 
The method of calculating the evaluation value will be described in detail 
later. 

Since rough search is performed using the two-dimensional images, 
search in a very narrow scope is enough for the three-dimensional data. 
Therefore, the amount of calculation in total can be reduced as compared 
with the search utilizing the three-dimensional data only, whereby patch 
up operation can be done at high speed. 

Thereafter, whether the calculated evaluation value is within a 
prescribed value is determined (step #306). When it is not larger than the 
prescribed value, two three-dimensional images are converted to the same 
coordinate system by using the last calculated coordinate conversion 
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parameter for patching up (step #307) and the operation is completed (step 
#310). The method of coordinate conversion will be described later. 

When the evaluation value is larger than the prescribed value in 
step #306, then whether the number of repetition of continuity evaluation 
is not smaller than a prescribed number is determined (step #308). If it is 
not smaller than the number, the patch up operation in step #307 is 
performed. The reason is that when the number of repetition exceeds a 
prescribed number, the evaluation value converges, making further 
repetition unnecessary. 

When the number of repetition is smaller than the prescribed 
number in step #308, then the angle of rotation of the camera is slightly 
changed so that the evaluation value calculated in step #305 becomes 
smaller (step #309). Thereafter, the flow proceeds to step #304 and 
continuity evaluation of the patches at the junction portion is repeated. 

In the following, each of the steps will be described in greater detail. 

Firstly, the searching method of the junction point from two- 
dimensional color images in step #302 is described with reference to Figs. 
14-16. The description is given on the premise that two images to be 
patched up have overlapping portions (having the width of T pixels) as 
shown in Fig. 14. Referring to Fig. 15 (a), a reference window is set at a 
central portion of the overlapping portion of one of the images (the dotted 
line in Fig. 15 (a) denotes the center line of the overlapping portion). Fig. 
15 (b) is an enlarged view of the reference window portion of Fig. 15 (a). 
This reference window is further divided into small windows each having 
the size of about 8x8 (pixels). Of the small windows, one having a 
complicated shape or complicated patterns (having large value of 
distribution) is used as a comparing window. The reason for this is that 
when a portion having clear edges or complicated patterns or shapes is 
used, reliability of evaluation can be improved. 

On the other image, a searching window which has the same size as 
the reference window and which is movable to move on the entire 
overlapping portion is set (Fig. 16). 

In this searching window, small windows are provided at relatively 
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the same positions as the comparing windows in the reference window. 
The square sum of the difference in luminance between the small window 
and the comparing window is used as the evaluation value, and the 
junction point is searched. 

Next, the method for calculating a camera rotation angle based on 
the two-dimensional image junction point, the focal length and the distance 
to the reference plane of step #303 is described. 

When we represent the pixel size as PS, camera plane size as 2 x S, 
focal length as f, and the number of shifted pixels as T, the camera rotation 
angle G can be obtained by the following equation, if the axes of rotation 
and the camera position coincide with each other (that is, the rotation axis 
intersects the optical axis of the camera) (Fig. 17)- 

9 = n - arctan (S / £) - arctan ((S - PS x t) / £) 

If the rotation axis does not coincide with the camera position (when 
the rotation axis and the camera axis are deviated from each other), the 
following relation holds, where r represents radius of rotation (distance 
between the rotation axis and the optical axis of the camera), and D 
represents the distance to the reference plane- 

t x PS x D/f = 2S x D/f - (D + r x sin 9) / tan(rc - arctan (f/S) - 9) - S x 
D/f - r x cos 9 

When the rotation axis and the camera position do not coincide with 
each other, the calculation becomes very complicated and the angle of 
rotation cannot be obtained easily. Therefore, it is preferable to provide a 
table showing the number of pixels (t) and the corresponding angles 
obtained by searching, so that the angle of rotation can readily be found. 

The method for calculating the coordinate conversion parameter of 
the camera in step #304, and the method of coordinate conversion of step 
#307 are now described. When we represent the coordinate systems of two 
cameras as CI (XI, Yl, Zl) and C2 (X2, Y2, Z2), the position of the camera 
rotation axis as T (tl, t2, t3) and the direction of rotation of the camera as 
(l, 0, 0) (rotation about the X axis in the coordinate system of Cl), then C2 
when rotated by 9 about the axis of rotation is converted to the coordinate 
system of Cl in accordance with the following equation' 
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C2-T = R(9)#(C1-T) 
where R (0) is obtained by the following equation, based on the angle 0 of 
camera rotation' 



R(0) = 



1 0 0 
0 cos0 -sin0 
0 sin0 cos0 



Therefore, conversion of the C2 coordinate system to the Cl 
coordinate system can be represented by the following equation, using 
parameters R (0) and T- 

C1=R(0)- 1 *(C2-T)+T 

More specifically, the point Cl (of Cl coordinate system) is moved in 
parallel onto the rotation axis, the coordinate is converted to the C2 
coordinate system on the rotation axis (rotated by 0), and the point is 
moved in parallel from the rotation axis to the point C2. 

The above-described operation is for the panning angle. Similar 
coordinate conversion can be performed by using the Y axis as the rotation 
axis for the tilting angle. 

The method for calculating the continuity evaluation value of the 
patches at the junction portion in step #305 is now described in greater 
detail with reference to the flowchart of Fig. 18 and to Fig. 19. 

Firstly, three-dimensional data of two images including the point of 
junction searched for from the two-dimensional color images are taken (step 
#401). Thereafter, normals at the center of the planes 1-12 at the portion 
of junction between the first image (the image represented by the white 
circle in Fig. 19) and the second image (the image represented by the block 
circle in Fig. 19) are calculated (step #402). 

Then, for the first image, 

el(l) = (angle formed by the normals of 1 and 2-1) - (angle formed 
by the normals of 1 and 2-12) 

is calculated. Similar calculation is performed for n sets of planes 
following the plane 4, and square sum (El) of the results is obtained (step 
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#403). 

For the second image, 

e2 (l) = (angle formed by the normals of 3 and 2-2) - (angle formed 
by the normals of 3 and 2-12) 

is calculated. Similar calculation is performed for n sets of planes 
following the plane 6, and the square sum (E2) of the results is obtained 
(step#404). 

Then, whether a smooth junction is obtained or not is evaluated by 

using 

(El +E2) /n 

(step #405), and the evaluation value is returned to the main routine (step 
#406). 

Patch up of the data photographed by a plurality of cameras is now 
described. When a plurality of cameras are used for photographing, the 
relative position and orientation can be measured by sensing the cameras 
by each other. Therefore, based on the data, the position of the object 
(corresponding to the position of the rotation axis) and the angle between 
the cameras viewed from the object (corresponding to the angle of rotation) 
are calculated. Based on these calculated values, coordinate conversion 
parameters are calculated, and two three-dimensional images are converted 
to the same coordinate system and patched up. The details of the patching 
up operation are similar to those when the camera frame is used described 
above. Therefore, description thereof is not repeated here. 

When the cameras are photographed by each other, the camera 
position can be calculated with higher precision if a lens having longer focal 
length than used for sensing the object is used. By doing so, undesirable 
influence on the object data at the time of patching can be avoided. 

Next, patch up of data when the object is photographed placed on a 
rotary stage is described with reference to Fig. 20 and the flowchart of Fig. 
21. Fig. 20 shows the rotary stage on which the object is placed. The 
rotary stage has polygonal circumference. The normal of each plane is 
orthogonal to the rotation axis, and the planes are arranged at equal 
distance from the rotary axis. Therefore, when each plane is measured, 



the distance and position of the rotary axis of the rotary stage can be 
calculated. 

For example, four three-dimensional and two-dimensional data are 
photographed (step #502) by rotating the rotary stage by 90 degrees for 
every sensing operation, such that the rotary stage is within the 
measurement scope as shown in Fig. 22 (which is a model of sensing 
operation using the rotary stage). Thereafter, with respect to the data of 
the photographed four images, the data of the object and the data of the 
rotary stage portion are separated from each other, and a group of planes, 
which is lower part of the rotary stage, is extracted (step #503). At this 
time, the plane portion of the rotary stage may have a specific color, to 
further facilitate extraction with the color image. 

Thereafter, using the data of the rotary stage portion from among the 
data separated in step #503, the position and attitude of the rotary stage 
are calculated (step #504). This method will be described in greater detail 
later. 

Based on the position and attitude of the rotary stage and the angle 
of rotation calculated in the subroutine of step #504, coordinate conversion 
parameter (about the rotation axis) for each photographed data is 
calculated (step #505). Based on the parameter, coordinate conversion is 
performed, whereby respective photographed data are integrated in one 
coordinate system (step #506). The method of calculating the parameter 
from the rotation angle and the method of coordinate conversion are the 
same as the method of calculating the parameter and method of coordinate 
conversion when the above-described camera universal head is used, except 
that the angle of rotation of the camera is replaced by the angle of rotation 
of the rotary stage. 

Thereafter, the junction portion is set (the method will be described 
later), data out of the scope of each photographed data is deleted, a plane is 
re -constructed at the junction portion (step #507), and patching up of the 
three-dimensional data is completed (step #5015). 

As a result, as shown in Fig. 23, the first image (represented by the 
black points) and the second image (white points) are patched up at the 
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boundary, thus resulting in one image. 

The method for calculating the position and the attitude of the rotary 
stage in step #504 is now described with reference to the flowchart of Fig. 
24. Firstly, three-dimensional data of the rotary stage and the color image 
of the rotary stage are taken (step #601). The data is divided for each 
plane (step #602). Thereafter, normal vector of the plane is calculated for 
every plane (step #603). A line that is orthogonal to the normal vector and 
at an equal distance from respective plane is defined as the rotation axis 
(step #604), and the rotation axis is returned to the main routine as the 
position and attitude of the rotary stage (step #605). 

The method for setting the junction portion is now described in 
greater detail. 

Firstly, the case where the real data photographed is not changed is 
described with reference to the flowchart of Fig. 25. Firstly, of the data cut 
at the boundary of the images (four planes including the axis of the rotary 
stage and orthogonal to each other), only that data which is sandwiched by 
two boundaries of images is regarded as effective data, and other data are 
canceled (step #701). Correspondence between end points of the 
photographed data is determined (two points which are close to each other 
are regarded as corresponding points), the images are patched up 
successively (step #702), and the flow returns to the main routine (step 
#703). 

In this case, when the data are canceled, overlapping portions may 
be left for two images to be patched up. By doing so, patching up can be 
performed smoothly by searching for the junction point, as already 
described with reference to the patching up operation using a universal 
head for the camera. 

The case where photographed data are changed for smooth patch up 
operation is now described with reference to the flowchart of Fig. 26 and to 
Fig. 27. With respect to photographed data, points approximately at the 
same distance from the boundary between the images (four planes 
including the axis of the rotary stage and orthogonal to each other) are 
determined as corresponding points (l-l and 2-1, 1-2 and 2-2, 1-n and 
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2-n of Fig. 27) (step #801), and a new point (the point marked by x in Fig. 
27) is generated based on two points corresponding to the data up to a 
prescribed distance from the boundary (step #802). The new point that is 
to be generated is determined in the following manner, in accordance with 
the distance from the boundary. 

When we represent a prescribed scope (in which the new point is 
generated) from the boundary as D, the point of one photographed data as 
XI, the point of another photographed data as X2, average distance from 
the boundary to the two points (XI, X2) as d, and the newly generated point 
as X3, then the following relation holds : 

X3 = (CD+d) x XI + (D-d) x X2) / (2 x D) 

The plane is re -constructed by using the newly generated data near 
the boundary (the scope whose distance from the boundary is up to D) and 
by using real data at other portions (the scope whose distance from the 
boundary exceeds D) (step #803), and the flow returns to the main routine 
(step #804). 

Further, by applying a recess/projection at every 90 degrees on the 
rotary stage as shown in Fig. 20 (b), the angle of rotation can be made very 
precise at extremely low cost. By performing coordinate conversion based 
on the axis of rotation calculated in advance for the four images 
photographed with the stage rotated, the entire peripheral data can be 
obtained. When such a rotary stage is used, the object can be set at an 
arbitrary position within the scope of measurement, and therefore the 
sensing operation is much facilitated. 

The zoom patch up method when a zooming input is provided as 
described above is now explained with reference to the flowchart of Fig. 28. 

Firstly, photographed data having different magnifications are taken 
in accordance with the method described above in conjunction with the 
zooming input (step #901). Then, patch up of the images using the camera 
universal head is performed in the same manner as shown in the flowchart 
of Fig. 13 described above (except for the last patch up operation), and 
calculation of the parameter for coordinate conversion and extraction of 
data at the boundary portion are performed (step #902). Here, before 
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searching for the junction point from two-dimensional color images (step 
#302 of Fig. 13), re-sampling is performed for the two-dimensional images 
and for the three-dimensional images (the method will be described later). 

Thereafter, coordinate conversion is performed with respect to the 
data having high magnification, the data having high magnification is 
integrated to the coordinate system of data having low magnification (step 
#903), a plane is re -constructed at the boundary portion (step #904), and 
the patch up operation is completed (step #905). 

Fig. 29 is a model of the zoom patch up operation. Firstly, data 
(having magnification of N2) of Fig. 29 (b) is re-sampled so that it comes to 
have the magnification of Nl as shown in Fig. 29 (c), and then re-sampled 
data is patched up with the data (having the magnification of Nl) of Fig. 29 
(a). Thereafter, the portion having had the magnification of N2 is 
returned to have the original magnification (N2), and as a result, a patched 
up image such as shown in Fig. 29 (d) is obtained. 

The re-sampling method of the two-dimensional and three- 
dimensional images is now described in greater detail. 

Firstly, the method for the two-dimensional image is described with 
reference to Fig. 30. 

In Fig. 30, the image represented by the solid lines is the image 
having the magnification of Nl, while the image represented by the dotted 
lines is the image having the magnification of N2 (in both images, the 
minimum square corresponds to one pixel, where Nl < N2). 

Re-sampling is performed for the image having the magnification of 
N2. The phase is matched so that the pixel at the upper left end of the 
image having the magnification N2 coincides with a sampling point of the 
image having the magnification of Nl. 

The re -sampling value (average brightness) is calculated by using a 
weighted mean value of the area of the pixels of the image having 
magnification of N2 included in the pixels of the image having the 
magnification of Nl. More specifically, the product of the brightness and 
area of the image having the magnification of N2 included in 1 pixel of the 
image having the magnification Nl are all added, and the result is divided 
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by the area of one pixel of the image having the magnification of Nl. 

The operation for the three-dimensional image is now described with 
reference to Fig. 31. Fig. 31 is a representation viewed from the camera. 

In Fig. 31, the image represented by the solid lines and the white 
circles is the image having the magnification of Nl, while the image 
represented by the dotted lines and the black circles is the image having 
the magnification of N2 (in both images, the minimum square represents 1 
pixel, Nl < N2). 

Re-sampling is performed on the image having the magnification of 
N2. The phase is matched such that the pixel at the upper left end of the 
image having the magnification of N2 coincides with a sampling point of 
the image having the magnification of Nl. 

The re-sampling value is calculated by using an intersection between 
the line of sight of the camera passing through the point of the image 
having the magnification of Nl, and a two-dimensional curved plane 
consisting of four points of the image having the magnification of N2 
surrounding the relevant point. 

In the above-described embodiment, calculation of the parameters for 
coordinate conversion is performed using both the two-dimensional color 
image and three-dimensional data. However, the coordinate conversion 
parameters can be calculated by using only the three-dimensional data, 
without searching for the junction point from the two-dimensional color 
image. Though a three-dimensional input has been described in the 
present embodiment, this invention can be similarly applied to the two- 
dimensional image input. 

[Effects of the Invention] 

As described above, according to the image input camera of the 
present invention, data of a plurality of continuous images are input, and 
relative positions of the input images are calculated to perform patch up 
thereof. Accordingly, even in a rough input environment (using a hand- 
held camera, for example), it is possible to obtain an image that has only a 
small error in patch up and thus has almost no awkward appearance at the 
patched up portion. 
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[Brief Description of the Drawings] 

Fig. 1 illustrates the principle of the light-section method. 

Fig. 2 is a schematic block diagram of the entire apparatus in 
accordance with the present invention. 

Fig. 3 is a perspective view showing a schematic structure of the 
entire apparatus in accordance with the present invention. 

Fig. 4 illustrates the incident range and the scan range of the 
reflected light entering the photographing device. 

Fig. 5 illustrates the light intensity distribution generated at the 
plane of the object. 

Fig. 6 illustrates the light intensity distribution generated at the 
light receiving plane of the photographing device. 

Fig. 7 illustrates the light intensity distribution generated at the 
light receiving plane of the photographing device. 

Fig. 8 illustrates the image patch up function. 

Fig. 9 is a flowchart illustrating an operation of the image patch up 
function. 

Fig. 10 illustrates a display for the image patch up function. 

Fig. 11 is a flowchart illustrating an operation of the partial zooming 
patch up function. 

Fig. 12 illustrates a camera model when photographing is performed 
using a camera universal head. 

Fig. 13 is a flowchart illustrating an operation for patching up three- 
dimensional data photographed by using the camera universal head. 

Fig. 14 illustrates the overlapping portions for patching up two- 
dimensional images. 

Fig. 15 illustrates a reference window for patching up two- 
dimensional images. 

Fig. 16 illustrates a search window for patching up two-dimensional 
images. 

Fig. 17 illustrates a calculation method of a camera rotation angle. 
Fig. 18 is a flowchart illustrating an operation of continuity 
evaluation of patches at the junction portion. 
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Fig. 19 illustrates patch up of data photographed by using the 
camera universal head. 

Fig. 20 is a perspective view showing an appearance of a rotary stage. 

Fig. 21 is a flowchart illustrating an operation for patching up three- 
dimensional data photographed by using the rotary stage. 

Fig. 22 illustrates photographing and image patch up operations 
using the rotary stage. 

Fig. 23 illustrates patch up of the data photographed by using the 
rotary stage. 

Fig. 24 is a flowchart illustrating an operation for calculating 
position and attitude of the rotary stage. 

Fig. 25 is a flowchart illustrating an operation of the setting method 
of the junction portion (without changing the real data). 

Fig. 26 is a flowchart illustrating an operation of the setting method 
of the junction portion (by changing the real data). 

Fig. 27 illustrates data generation points in the setting method of the 
junction portion (with the change of the real data). 

Fig. 28 is a flowchart illustrating an operation for patching up three- 
dimensional data photographed with zooming, using the camera universal 
head. 

Fig. 29 illustrates patch up of data photographed with zooming. 
Fig. 30 illustrates re-sampling of two-dimensional images. 
Fig. 31 illustrates re-sampling of three-dimensional images. 
[Description of the Reference Characters] 

#301- input means; #502- input means; #304: calculating means; 
#505: calculating means; #307: patch up means; and #506: patch up means. 
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[Document Name] Abstract 
[Abstract] 

[Subject] To perform image patch up at a relatively low cost, by inputting 
a plurality of pieces of image data from multiple viewpoints, calculating 
relative positions of the input images, and obtaining a conversion 
parameter of the image data for performing the patch up. 
[Solving Means] The image input camera of the present invention 
includes input means for inputting image data of an object, means for 
calculating, from a plurality of pieces of the image data input from multiple 
viewpoints, a coordinate conversion parameter for patch up of the plurality 
of pieces of the image data, and means for patching up the plurality of 
pieces of the image data. 
[Selected Drawing] Fig. 13 
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